Method and system for inter-prediction in decoding of video data

ABSTRACT

Embodiments of a method and system for inter-prediction in decoding video data are described herein. In various embodiments, a high-compression-ratio codec (such as H.264) is part of the encoding scheme for the video data. Embodiments pre-process control maps that were generated from encoded video data, and generating intermediate control maps comprising information regarding decoding the video data. The control maps indicate which units of video data in a frame are to be processed using an inter-prediction operation. In an embodiment, inter-prediction is performed on a frame basis such that inter-prediction is performed on an entire frame at one time. In other embodiments, processing of different frames is interleaved. Embodiments increase the efficiency of the inter-prediction such as to allow decoding of high-compression-ratio encoded video data on personal computers or comparable equipment without special, additional decoding hardware.

TECHNICAL FIELD

The invention is in the field of decoding video data that has beenencoded according to a specified encoding format, and more particularly,decoding the video data to optimize use of data processing hardware.

BACKGROUND

Digital video playback capability is increasingly available in all typesof hardware platforms, from inexpensive consumer-level computers tosuper-sophisticated flight simulators. Digital video playback includesdisplaying video that is accessed from a storage medium or streamed froma real-time source, such as a television signal. As digital videobecomes nearly ubiquitous, new techniques to improve the quality andaccessibility of the digital video are being developed. For example, inorder to store and transmit digital video, it is typically compressed orencoded using a format specified by a standard. Recently H.264, a videocompression scheme, or codec, has been adopted by the Motion PicturesExpert Group (MPEG) to be the video compression scheme for the MPEG-4format for digital media exchange. H.264 is MPEG-4 Part 10. H.264 wasdeveloped to address various needs in an evolving digital media market,such as relative inefficiency of older compression schemes, theavailability of greater computational resources today, and theincreasing demand for High Definition (HD) video, which requires theability to store and transmit about six times as much data as requiredby Standard Definition (SD) video.

H.264 is an example of an encoding scheme developed to have a muchhigher compression ratio than previously available in order toefficiently store and transmit higher quantities of video data, such asHD video data. For various reasons, the higher compression ratio comeswith a significant increase in the computational complexity required todecode the video data for playback. Most existing personal computers(PCs) do not have the computational capability to decode HD video datacompressed using high compression ratio schemes such as H.264.Therefore, most PCs cannot playback highly compressed video data storedon high-density media such as optical Blu-ray discs (BD) or HD-DVDdiscs. Many PCs include dedicated video processing units (VPUs) orgraphics processing units (GPUs) that share the decoding tasks with thePC. The GPUs may be add-on units in the form of graphics cards, forexample, or integrated GPUs. However, even PCs with dedicated GPUstypically are not capable of BD or HD-DVD playback. Efficient processingof H.264/MPEG-4 is very difficult in a multi-pipeline processor such asa GPU. For example, video frame data is arranged in macro blocksaccording to the MPEG standard. A macro block to be decoded hasdependencies on other macro blocks, as well as intrablock dependencieswithin the macro block. In addition, edge filtering of the edges betweenblocks must be completed. This normally results in algorithms thatsimply complete decoding of each macro block sequentially, whichinvolves several computationally distinct operations involving differenthardware passes. This results in failure to exploit the parallelism thatis inherent in modern day processors such as multi-pipeline GPUs.

One approach to allowing PCs to playback high-density media is theaddition of separate decoding hardware and software. This decodinghardware and software is in addition to any existing graphics card(s) orintegrated GPUs on the PC. This approach has various disadvantages. Forexample, the hardware and software must be provided for each PC which isto have the decoding capability. In addition, the decoding hardware andsoftware decodes the video data without particular consideration foroptimizing the graphics processing hardware which will display thedecoded data.

It would be desirable to have a solution for digital video data thatallows a PC user to playback high-density media such as BD or HD-DVDwithout the purchase of special add-on cards or other hardware. It wouldalso be desirable to have such a solution that decodes the highlycompressed video data for processing so as to optimize the use of thegraphics processing hardware, while minimizing the use of the CPU, thusincreasing speed and efficiency.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system with graphics processingcapability according to an embodiment.

FIG. 2 is a block diagram of elements of a GPU according to anembodiment.

FIG. 3 is a diagram illustrating a data and control flow of a decodingprocess according to an embodiment.

FIG. 4 is another diagram illustrating a data and control flow of adecoding process according to an embodiment.

FIG. 5 is a diagram illustrating a data and control flow of aninter-prediction process according to an embodiment.

FIGS. 6A, 6B, and 6C are diagrams of a macro block divided intodifferent blocks according to an embodiment.

FIG. 7 is a block diagram illustrating intra-block dependenciesaccording to an embodiment.

FIG. 8 is a diagram illustrating a data and control flow of anintra-prediction process according to an embodiment.

FIG. 9 is a block diagram of a frame after inter-prediction andintra-prediction have been performed according to an embodiment.

FIGS. 10A and 10B are block diagrams of macro blocks illustratingvertical and horizontal deblocking, which are performed on each macroblock according to an embodiment.

FIGS. 11A, 11B, 11C, and 11D show the pels involved in verticaldeblocking for each vertical edge in a macro block according to anembodiment.

FIGS. 12A, 12B, 12C, and 12D show the pels involved in horizontaldeblocking for each horizontal edge in a macro block according to anembodiment.

FIG. 13A is a block diagram of a macro block that shows vertical edges0-3 according to an embodiment.

FIG. 13B is a block diagram that shows the conceptual mapping of theshaded data from FIG. 13A into a scratch buffer according to anembodiment.

FIG. 14A is a block diagram that shows multiple macro blocks and theiredges according to an embodiment.

FIG. 14B is a block diagram that shows the mapping of the shaded datafrom FIG. 14A into the scratch buffer according to an embodiment.

FIG. 15A is a block diagram of a macro block that shows horizontal edges0-3 according to an embodiment.

FIG. 15B is a block diagram that shows the conceptual mapping of theshaded data from FIG. 15A into the scratch buffer according to anembodiment.

FIG. 16A is a bock diagram that shows multiple macro blocks and theiredges according to an embodiment.

FIG. 16B is a block diagram that shows the mapping of the shaded datafrom FIG. 16A into the scratch buffer according to an embodiment.

FIG. 17A is a bock diagram that shows multiple macro blocks and theiredges according to an embodiment.

FIG. 17B is a block diagram that shows the mapping of the shaded datafrom FIG. 17A into the scratch buffer according to an embodiment.

FIG. 18A is a bock diagram that shows multiple macro blocks and theiredges according to an embodiment.

FIG. 18B is a block diagram that shows the mapping of the shaded datafrom FIG. 18A into the scratch buffer according to an embodiment.

FIG. 19A is a bock diagram that shows multiple macro blocks and theiredges according to an embodiment.

FIG. 19B is a block diagram that shows the mapping of the shaded datafrom FIG. 19A into the scratch buffer according to an embodiment.

FIG. 20 is a block diagram of a source buffer at the beginning of adeblocking algorithm iteration according to an embodiment.

FIG. 21 is a block diagram of a target buffer at the beginning of adeblocking algorithm iteration according to an embodiment.

FIG. 22 is a block diagram of the target buffer after the left sidefiltering according to an embodiment.

FIG. 23 is a block diagram of the target buffer after the verticalfiltering according to an embodiment.

FIG. 24 is a block diagram of a new target buffer after a copy accordingto an embodiment.

FIG. 25 is a block diagram of the target buffer after a pass accordingto an embodiment.

FIG. 26 is a block diagram of the target buffer after a pass accordingto an embodiment.

FIG. 27 is a block diagram of the target buffer after a copy accordingto an embodiment.

The drawings represent aspects of various embodiments for the purpose ofdisclosing the invention as claimed, but are not intended to be limitingin any way.

DETAILED DESCRIPTION

Embodiments of a method and system for layered decoding of video dataencoded according to a standard that includes a high-compression ratiocompression scheme are described herein. The term “layer” as used hereinindicates one of several distinct data processing operations performedon a frame of encoded video data in order to decode the frame. Thedistinct data processing operations include, but are not limited to,motion compensation and deblocking. In video data compression, motioncompensation typically refers to accounting for the difference betweenconsecutive frames in terms of where each section of the former framehas moved to. In an embodiment, motion compensation is performed usinginter-prediction and/or intra-prediction, depending on the encoding ofthe video data.

Prior decoding methods performed all of the distinct data processingoperations on a unit of data within the frame before moving to a nextunit of data within a frame. In contrast, embodiments of the inventionperform a layer of processing on an entire frame at one time, and thenperform a next layer of processing. In other embodiment, multiple framesare processed in parallel using the same algorithms described below. Theencoded data is pre-processed in order to allow layered decoding withouterrors, such as errors that might result from processing interdependentdata in an incorrect order. The pre-processing prepares various sets ofencoded data to be operated on in parallel by different processingpipelines, thus optimizing the use of the available graphics processinghardware and minimizing the use of the CPU.

FIG. 1 is a block diagram of a system 100 with graphics processingcapability according to an embodiment. The system 100 includes a videodata source 112. The video data source 112 may be a storage medium suchas a Blu-ray disc or an HD-DVD disc. The video data source may also be atelevision signal, or any other source of video data that is encodedaccording to a widely recognized standard, such as one of the MPEGstandards. Embodiments of the invention will be described with referenceto the H.264 compression scheme, which is used in the MPEG-4 standard.Embodiments provide particular performance benefits for decoding H.264data, but the invention is not so limited. In general, the particularexamples given are for thorough illustration and disclosure of theembodiments, but no aspects of the examples are intended to limit thescope of the invention as defined by the claims.

System 100 further includes a central processing unit (CPU)-basedprocessor 108 that receives compressed, or encoded, video data 109 fromthe video data source 112. The CPU-based processor 108, in accordancewith the standard governing the encoding of the data 109, processes thedata 109 and generates control maps 106 in a known manner. The controlmaps 106 include data and control information formatted in such a way asto be meaningful to video processing software and hardware that furtherprocesses the control maps 106 to generate a picture to be displayed ona screen. In an embodiment, the system 100 includes a graphicsprocessing unit (GPU) 102 that receives the control maps 106. The GPU102 may be integral to the system 100. For example, the GPU 102 may bepart of a chipset made for inclusion in a personal computer (PC) alongwith the CPU-based processor 108. Alternatively, the GPU 102 may be acomponent that is added to the system 100 as a graphics card or videocard, for example. In embodiments described herein, the GPU 102 isdesigned with multiple processing cores, also referred to herein asmultiple processing pipelines or multiple pipes. In an embodiment, themultiple pipelines each contain similar hardware and can all be runsimultaneously on different sets of data to increase performance. In anembodiment, the GPU 102 can be classed as a single instruction multipledata (SIMD) architecture, but embodiments are not so limited.

The GPU 102 includes a layered decoder 104, which will be described ingreater detail below. In an embodiment, the layered decoder 104interprets the control maps 106 and pre-processes the data and controlinformation so that processing hardware of the GPU 102 can optimallyperform parallel processing of the data. The GPU 102 thus performshardware-accelerated video decoding. The GPU 102 processes the encodedvideo data and generates display data 115 for display on a display 114.The display data 115 is also referred to herein as frame data or decodedframes. The display 114 can be any type of display appropriate to aparticular system 100, including a computer monitor, a televisionscreen, etc.

In order to facilitate describing the embodiments, an overview of thetype of video data that will be referred to in the description nowfollows. A SIMD architecture is most effective when it conductsmultiple, massively parallel computations along substantially the samecontrol flow path. In the examples described herein, embodiments of thelayered decoder 104 include an H.264 decoder running GPU hardware tominimize the flow control deviation in each shader thread. A shader asreferred to herein is a software program specifically for renderinggraphics data or video data as known in the art. A rendering task mayuse several different shaders.

The following is a brief explanation of some of the terminology used inthis description.

A luma or chroma 8-bit value is called a pel. All luma pels in a frameare named in the Y plane. The Y plane has a resolution of the picturemeasured in pels. For example, if the picture resolution is said to be720×480, the Y plane has 720×480 pels. Chroma pels are divided into twoplanes: a U plane and a V plane. For purposes of the examples used todescribe the embodiments herein, a so-called 420 format is used. The 420format uses U and V planes having the same resolution, which is half ofthe width and height of the picture. In a 720×480 example, the U and Vresolution is 360×240 measured in pels.

Hardware pixels are pixels as they are viewed by the GPU on the readfrom memory and the write to the memory. In most cases this is a4-channel, 8-bit per channel pixel commonly known as RGBA or ARGB.

As used herein, “pixel” also denotes a 4×4 pel block selected as a unitof computation. It means that as far as the scan converter is concernedthis is the pixel, causing the pixel shader to be invoked per each 4×4block. In an embodiment, to accommodate this view, the resolution of thetarget surface presented to the hardware is defined as one quarter ofthe width and of the height of the original picture resolution measuredin pels. For example, returning to the 720×480 picture example, theresolution of the target is 180×120.

The block of 16×16 pels, also referred to as a macro block, is themaximal semantically unified chunk of video content, as defined by MPEGstandards. A block of 4×4 pels is the minimal semantically unified chunkof the video content.

There are 3 different physical target picture or target frame layoutsemployed depending on the type of the picture being decoded. The targetframe layouts are illustrated in Tables 1-3.

Let PicWidth be the width of the picture in pels (which is the same asbytes) and PicHeight be the height of the picture in scan lines (forexample, 720×480 in the previous example. Table 1 shows the physicallayout based on the picture type.

TABLE 1 Field Frame/AFF Even Odd Y {0, 0}, {PicWidth− {0, 0}, {PicWidth−{0, PicHeight/2}, 1, PicHeight−1} 1, PicHeight/2−1} {PicWidth−1,PicHeight} U {0, PicHeight}, {PicWidth/ {0, PicHeight}, {PicWidth/ {0,5 * PicHeight/4}, {PicWidth/ 2−1, 3 * PicHeight/2−1} 2−1, 5 *PicHeight/4−1} 2−1, 3 * PicHeight/2−1} V {PicWidth/2, PicHeight},{PicWidth/2, PicHeight}, {PicWidth/2, 5 * PicHeight/4}, {PicWidth−{PicWidth− {PicWidth− 1, 3 * PicHeight/2−1} 1, 5 * PicHeight/4−1} 1, 3 *PicHeight/2−1}

Following Tables 2 and 3 are visual representations of Table 1 for aframe/AFF picture and for a field picture, respectively.

TABLE 2 Frame/AFF picture Y plane U plane V plane

TABLE 3 Field picture Y plane even Y plane odd U plane even V plane evenU plane odd V plane odd

The field type picture keeps even and odd fields separately until a last“interleaving” pass. The AFF type picture keeps field macro blocks astwo complimentary pairs until the last “interleaving” pass. Theinterleaving pass interleaves even and odd scan lines and builds oneprogressive frame.

Embodiments described herein include a hardware decoding implementationof the H.264 video standard. H.264 decoding contains three major parts:inter-prediction; intra-prediction; and deblocking filtering. In variousembodiments, inter-prediction and intra-prediction are also referred toas motion compensation because of the effect of performinginter-prediction and intra-prediction.

According to embodiments a decoding algorithm consists of three“logical” passes. Each logical pass adds another layer of data onto thesame output picture or frame. The first “logical” pass is theinter-prediction pass with added inversed transformed coefficients. Thefirst pass produces a partially decoded frame. The frame includes macroblocks designated by the encoding process to be decoded using eitherinter-prediction or intra-prediction. Because only the inter-predictionmacro blocks are decoded in the first pass, there will be “holes” or“garbage” data in place of intra-prediction macro blocks.

A second “logical” pass touches only intra-prediction macro blocks leftafter the first pass is complete. The second pass computes theintra-prediction with added inversed transformed coefficients.

A third pass is a deblocking filtering pass, which includes a deblockcontrol map generation pass. The third pass updates pels of the samepicture along the sub-block (e.g., 4×4 pels) edges.

The entire decoding algorithm as further described herein does notrequire intervention by the host processor or CPU. Each logical pass mayinclude many physical hardware passes. In an embodiment, all of thepasses are pre-programmed by a video driver, and the GPU hardware movesfrom one pass to another autonomously.

FIG. 2 is a block diagram of elements of a GPU 202 according to anembodiment. The GPU 202 receives control maps 206 from a source such asa host processor or host CPU. The GPU 202 includes a video driver 222which, in an embodiment, includes a layered decoder 204. The GPU 202also includes processing pipelines 220A, 220B, 220C, and 220D. Invarious embodiments, there could be less than four or more than fourpipelines 220. In other embodiments, more than one GPU 202 may becombined to share processing tasks. The number of pipelines is notintended to be limiting, but is used in this description as a convenientnumber for illustrating embodiments of the invention. In manyembodiments, there are significantly more than four pipelines. As thenumber of pipelines is increased, the speed and efficiency of the GPU isincreased.

An advantage of the embodiments described is the flexibility and ease ofuse provided by the layered decoder 204 as part of the driver 222. Thedriver 222, in various embodiments, is software that can be downloadedby a user of an existing GPU to extend new layered decoding capabilityto the existing GPU. The same driver can be appropriate for all existingGPUs with similar architectures. Multiple drivers can be designed andmade available for different architectures. One common aspect of driversincluding layered decoders described herein is that they immediatelyallow efficient decoding of video data encoded using H.264 and similarformats by maximizing the use of available graphics processing pipelineson an existing GPU.

The GPU 202 further includes a Z-buffer 216 and a reference buffer 218.As further described below, Z buffer is used as control information, forexample to decide which macro blocks are processed and which are not inany layer. The reference buffer 218 is used to store a number of decodedframes in a known manner. Previously decoded frames are used in thedecoding algorithm, for example to predict what a next or subsequentframe might look like.

FIG. 3 is a diagram illustrating a flow of data and control in layereddecoding according to an embodiment. Control maps 306 are generated by ahost processor such as a CPU, as previously described. The control maps306 are generated according to the applicable standard, for exampleMPEG-4. The control maps 306 are generated on a per-frame basis. Acontrol map 306 is received by the GPU (as shown in FIGS. 1 and 2). Thecontrol maps 306 include various information used by the GPU to directthe graphics processing according to the applicable standard. Forexample, as previously described, the video frame is divided into macroblocks of certain defined sizes. Each macro block may be encoded suchthat either inter-prediction or intra-prediction must be used to decodeit. The decision to encode particular macro blocks in particular ways ismade by the encoder. One piece of information conveyed by the controlmaps 306 is which decoding method (e.g., inter-prediction orintra-prediction) should be applied to each macro block.

Because the encoding scheme is a compression of data, one of the aspectsof the overall scheme is a comparison of one frame to the next in timeto determine what video data does not change, and what video datachanges, and by how much. Video data that does not change does not needto be explicitly expressed or transmitted, thus allowing compression.The process of decoding, or decompression, according to the MPEGstandards, involves reading information in the control maps 306including this change information per unit of video data in a frame, andfrom this information, assembling the frame. For example, consider amacro block whose intensity value has changed from one frame to another.During inter-prediction, the decoder reads a residual from the controlmaps 306. The residual is an intensity value expressed as a number. Theresidual represents a change in intensity from one frame to the next fora unit of video data.

The decoder must then determine what the previous intensity value wasand add the residual to the previous value. The control maps 306 alsostore a reference index. The reference index indicates which previouslydecoded frame of up to sixteen previously decoded frames should beaccessed to retrieve the relevant, previous reference data. The controlmaps also store a motion vector that indicates where in the selectedreference frame the relevant reference data is located. In anembodiment, the motion vector refers to a block of 4×4 pels, butembodiments are not so limited.

The GPU performs preprocessing on the control map 306, including setuppasses 308, to generate intermediate control maps 307. The setup passes308 include sorting surfaces for performing inter-prediction for theentire frame, intra-prediction for the entire frame, and deblocking forthe entire frame, as further described below. The setup passes 308 alsoinclude intermediate control map generation for deblocking passesaccording to an embodiment. The setup passes 308 involve running“pre-shaders” that can be referred to as software programs of relativelysmall size (compared to the usual rendering shaders) to read the controlmap 306 without incurring the performance penalty for running the usualrendering shaders.

In general, the intermediate control maps 307 are the result ofinterpretation and reformulation of control map 306 data and controlinformation so as to tailor the data and control information to run inparallel on the particular GPU hardware in an optimized way.

In yet other embodiments, all the control maps are generated by the GPU.The initial control maps are CPU-friendly and data is arranged per macroblock. Another set of control maps can be generated from the initialcontrol maps using the GPU, where data is arranged per frame (forexample, one map for motion vectors, one map for residual).

After setup passes 308 generate intermediate control maps 307, shadersare run on the GPU hardware for inter-prediction passes 310. In somecases, inter-prediction passes 310 may not be available because theframe was encoded using intra-prediction only. It is also possible for aframe to be encoded using only inter-prediction. It is also possible fordeblocking to be omitted.

The inter-prediction passes are guided by the information in the controlmaps 306 and the intermediate control maps 307. Intermediate controlmaps 307 include a map of which macro blocks are inter-prediction macroblocks and which macro blocks are intra-prediction macro blocks.Inter-prediction passes 310 read this “inter-intra” information andprocess only the macro blocks marked as inter-prediction macro blocks.The intermediate control maps 307 also indicate which macro blocks orportions of macro blocks may be processed in parallel such that use ofthe GPU hardware is optimized. In our example embodiment there are fourpipelines which process data simultaneously in inter-prediction passes310 until inter-prediction has been completed on the entire frame. Inother embodiments, the solution described here can be scaled with thehardware such that more pipelines allow simultaneous processing of moredata.

When the inter-prediction passes 310 are complete, and there areintra-predicted macro blocks, there is a partially decoded frame 312.All of the inter-prediction is complete for the partially decoded frame312, and there are “holes” for the intra-prediction macro blocks. Insome cases, the frame may be encoded using only inter-prediction, inwhich case the frame has no “holes” after inter-prediction.

Intra-prediction passes 314 use the control maps 306 and theintermediate control maps 307 to perform intra-prediction on all of theintra-prediction macro blocks of the frame. The intermediate controlmaps 307 indicate which macro blocks are intra-prediction macro blocks.Intra-prediction involves prediction of how a unit of data will lookbased on neighboring units of data within a frame. This is in contrastto inter-prediction, which is based on differences between frames. Inorder to perform intra-prediction on a frame, units of data must beprocessed in an order that does not improperly overwrite data.

When the intra-prediction passes 314 are complete, there is a partiallydecoded frame 316. All of the inter-prediction and intra-predictionoperations are complete for the partially decoded frame 316, butdeblocking is not yet performed. Decoding on a macro block level causesa potentially visible transition on the edges between macro blocks.Deblocking is a filtering operation that smoothes these transitions. Inan embodiment, the intermediate control maps 307 include a deblockingmap (if available) that indicates an order of edge processing and alsoindicates filtering parameter. No deblocking map is available ifdeblocking is not required. In deblocking, the data from adjacent macroblock edges is combined and rewritten so that the visible transition isminimized. In an embodiment, the data to be operated on is written outto scratch buffers 322 for the purpose of rearranging the data to beoptimally processed in parallel on the hardware, but embodiments are notso limited.

After the deblocking passes 318, a completely decoded frame 320 isstored in the reference buffer (reference buffer 218 of FIG. 2, forexample). This is the reference buffer accessed by the inter-predictionpasses 310, as shown by arrow 330.

FIG. 4 is another diagram illustrating a flow 400 of data and control invideo data decoding according to an embodiment. FIG. 4 is anotherperspective of the operation illustrated in FIG. 3 with more detail.Control maps 406 are received by the GPU. In order to generate anintermediate control map that indicates which macro blocks are forinter-prediction, a comparison value in the Z-buffer is set to “inter”at 408. The comparison value can be a single bit that is set to “1” or“0”, but embodiments are not so limited. With the comparison value setto “inter”, a small shader, or “pre-shader” 410 is run on the controlmaps 406 to create the Z-buffer 412 and intermediate control maps 413.The Z-buffer includes information that tells an inter-prediction shader414 which macro blocks are to be inter-predicted and which are not. Inan embodiment this information is determined by Z-testing, butembodiments are not so limited. Macro blocks that are not indicated asinter-prediction macro blocks will not be processed by theinter-prediction shader 414, but will be skipped or discarded. Theinter-prediction shader 414 is run on the data using control informationfrom control maps 406 and an intermediate control map 413 to produce apartially decoded frame 416 in which all of the inter-prediction macroblocks are decoded, and all of the remaining macro blocks are notdecoded.

In another implementation, the Z buffer testing of whether a macro blockis an inter-prediction macro block or an intra-prediction macro block isperformed within the inter prediction shader 414.

The value set at 408 is then reset at 418 to indicate intra-prediction.In another embodiment, the value is not reset, but rather another bufferis used. A pre-shader 420 creates a Z-buffer 415 and intermediatecontrol maps 422. The Z-buffer includes information that tells anintra-prediction shader 424 which macro blocks are to be intra-predictedand which are not. In an embodiment this information is determined byZ-testing, but embodiments are not so limited. Macro blocks that are notindicated as intra-prediction macro blocks will not be processed by theintra-prediction shader 424, but will be skipped or discarded. The interintra-prediction shader 424 is run on the data using control informationfrom control maps 406 and an intermediate control map 422 to produce aframe 426 in which all of the inter-prediction macro blocks are decodedand all of the intra-prediction macro blocks are decoded. This is theframe that is processed in the deblocking operation.

Inter-Prediction

As previously discussed, inter-prediction is a way to use pels fromreference pictures or frames (future (forward) or past (backward)) topredict the pels of the current frame. FIG. 5 is a diagram illustratinga data and control flow of an inter-prediction process 500 for a frameaccording to an embodiment. In an embodiment, the geometrical mesh foreach inter-prediction pass consists of a grid of 4×4 rectangles in the Ypart of the physical layout and 2×2 rectangles in the UV part (16×16 or8×8 pels, where 16×16 pels is a macro block). A shader (in anembodiment, a vertex shader) parses the control maps for each macroblock's control information and broadcasts the preprocessed controlinformation to each pixel 502 (in this case, a pixel is a 4×4-block).The control information includes an 8-bit macro block header, multipleIT coefficients and their offsets, 16 pairs of motion vectors and 8reference frame selectors. Z-testing as previously described indicateswhether the macro block is not an inter-prediction block, in which case,its pixels will be “killed” or skipped from “rendering”.

At 504, a particular reference frame among various reference frames inthe reference buffer is selected using the control information. Then, at506, the reference pels within the reference frame are found. In anembodiment, finding the correct position of the reference pels insidethe reference frame includes computing the coordinates for each 4×4block. The input to the computation is the top-left address of thetarget block in pels, and the delta obtained from the proper controlmap. The target block is the destination block, or the block in theframe that is being decoded.

As an example of finding reference pels, let MvDx, MvDy be the deltaobtained from the control map. MvDx,MvDy are the x,y deltas computed inthe appropriate coordinate system. This is true for a frame picture andframe macro block of an AFF picture in frame coordinates, and for afield picture and field macro block of an AFF picture in the fieldcoordinate system of proper polarity. In an embodiment, the delta is thedelta between the X,Y coordinates of the target block and the X,Ycoordinates of the source (reference) block with 4-bit fractionalprecision.

When the reference pels are found, they are combined at 508 with theresidual data (also referred to as “the residual”) that is included inthe control maps. The result of the combination is written to thedestination in the partially decoded frame at 512. The process 500 is aparallel process and all blocks are submitted/executed in parallel. Atthe completion of the process, the frame data is ready forintra-prediction. In an embodiment, 4×4 blocks are processed in parallelas described in the process 500, but this is just an example. Otherunits of data could be treated in a similar way.

Intra-Prediction

As previously discussed, intra-prediction is a way to use pels fromother macro blocks or portions of macro blocks within a pictures orframe to predict the pels of the current macro block or portion of amacro block. FIGS. 6A, 6B, and 6C are diagrams of a macro block dividedinto different blocks according to an embodiment. FIG. 6A is a diagramof a macro block that includes 16×16 pels. FIG. 6B is diagram of 8×8blocks in a macro block. FIG. 6C is a diagram of 4×4 blocks in a macroblock. Various intra-prediction cases exist depending on the encodingperformed. For example, macro blocks in a frame may be divided intosub-blocks of the same size. Each sub-block may have from 8 cases to 14cases, or shader branches. The frame configuration is known beforedecoding from the control maps.

In an embodiment, a shader parses the control maps to obtain controlinformation for a macro block, and broadcasts the preprocessed controlinformation to each pixel (in this case, a pixel is a 4×4-block). Theinformation includes an 8-bit macro block header, a number of ITcoefficients and their offsets, availability of neighboring blocks andtheir types, and for 16×16 and 8×8 blocks, prediction values andprediction modes. Z-testing as previously described indicates whetherthe macro block is not an intra-prediction block, in which case, itspixels will be “killed” or skipped from “rendering”.

Dependencies exist between blocks because data from an encoded (not yetdecoded) block should not be used to intra-predict a block. FIG. 7 is ablock diagram that illustrates these potential intra-block dependencies.Sub-block 702 depends on its neighboring sub-blocks 704 (left), 706(up-left), 708 (up), and 710 (up-right).

To avoid interdependencies inside the macro block the 16 pixels inside a4×4 rectangle (Y plane) are rendered in a pass number indicated insidethe cell. The intra-prediction for a UV macro block and a 16×16 macroblock are processed in one pass. Intra-prediction for an 8×8 macro blockis computed in 4 passes; each pass computes the intra-prediction for one8×8 block from left to right and from top to bottom. Table 4 illustratesan example of ordering in a 4×4 case.

TABLE 4 0 1 2 3 2 3 4 5 4 5 6 7 6 7 8 9

To avoid interdependencies between the macro blocks the primitives(blocks of 4×4 pels) rendered in the same pass are organized into a listin a diagonal fashion.

Each cell below in Table 5 is a 4×4 (pixel) rectangle. The number insidethe cell connects rectangles belonging to the same lists. Table 5 is anexample for 16*8×16*8 in the Y plane:

TABLE 5 0 1 2 3 4 5 6 7 2 3 4 5 6 7 8 9 4 5 6 7 8 9 10 11 6 7 8 9 10 1112 13 8 9 10 11 12 13 14 15 10 11 12 13 14 15 16 17 12 13 14 15 16 17 1819 14 15 16 17 18 19 20 21

The diagonal arrangement keeps the following relation invariantseparately for Y, U and V parts of the target surface:

Frame/Field Picture:

if k is the pass number, k>0 && k<DiagonalLength−1, MbMU[2] arecoordinates of the macro block in the list, then MbMU[1]+MbMU[0]/2+1=k.

An AFF picture makes the process slightly more complex.

The same example as above with an AFF picture is illustrated in Table 6.

TABLE 6 0 2 4 6 8 10 12 14 1 3 5 7 9 11 13 15 4 6 8 10 12 14 16 18 5 7 911 13 15 17 19 8 10 12 14 16 18 20 22 9 11 13 15 17 19 21 23 12 14 16 1820 22 24 26 13 15 17 19 21 23 25 27

Inside all of the macro blocks, the pixel rendering sequence stays thesame as described above.

There are three types of intra predicted blocks from the perspective ofthe shader: 16×16 blocks, 8×8 blocks and 4×4 blocks. The driver providesan availability mask for each type of block. The mask indicates whichneighbor (upper, upper-right, upper-left or left is available). How themask is used depends on the block. For some blocks not all masks areneeded. For some blocks, instead of the upper-right masks, two leftmasks are used, etc. If the neighboring macro block is available, thepixels from it are used for the target block prediction according to theprediction mode provided to the shader by the driver.

There are two types of neighbors: upper (upper-right, upper, upper-left)and left.

The following describes computation of neighboring pel coordinates fordifferent temporal types of macro blocks of different picture typesaccording to an embodiment.

EvenMbXPU is a x coordinate of the complimentary pair of macro block

EvenMbYPU is a y coordinate of the complimentary pair of macro block

YPU is y coordinate of the current scan line.

MbXPU is a x coordinate of the macro block containing the YPU scan line

MbYPU is a y coordinate of the macro block containing the YPU scan line

MbYMU is a y coordinate of the same macro block in macro block units

MbYSzPU is a size of the macro block in Y direction.

Frame/Field Picture:

Function to compute x,y coordinates of pels in the neighboring macrobloc to the left:

XNeighbrPU=MbXPU−1

YNeighbrPU=YPU

Function to compute x,y coordinates of pels in the neighboring macrobloc to the up:

XNeighbrPU=MbXPU

YNeighbrPU=MbYPU−1;

AFF Picture:

Function to compute x,y coordinates of pels in the neighboring macrobloc to the left:

   EvenMbYPU = (MbYMU / 2) * 2    XNeighbrPU = MbXPU − 1   Frame->Frame:  Field ->Field:    YNeighbrPU = YPU    break;   Frame->Field:   //Interleave scan lines from even and odd neighboring field   macro block   YIsOdd = YPU%2    YNeighbrPU = EvenMbYPU + (YPU − EvenMbYPU)/2 +YIsOdd * MbYSzPU    break;   Field->Frame:   // Take only even or oddscan lines from the neighboring pair of frame macro blocks.    MbIsOdd =MbYMU % 2    YNeighbrPU = EvenMbYPU + (YPU − MbYPU)*2 + MbIsOdd

Function to compute x,y coordinates of pels in the neighboring macrobloc to the up:

  MbIsOdd = MbYMU % 2   XNeighbrPU = MbXPU   Frame -> Frame:   Frame ->Field:    YNeighbrPU = MbYPU − 1 − MbYSzPU * ( 1 − MbIsOdd);    break;  Field -> Field:    MbIsOdd = 1; // it allows always to elevate intothe macro block of the same polarity.   Field -> Frame:    YNeighbrPU =MbYPU − MbYSzPU * MbIsOdd +    MbIsOdd − 2 ;    break;

FIG. 8 is a diagram illustrating a data and control flow 800 of anintra-prediction process according to an embodiment. At 802, the layereddecoder parses the control map macro block header to determine types ofsubblocks within a macro block. The subblocks identified to be renderedin the same physical pass are assigned the same number “X” at 804. Toavoid interdependencies between macro blocks, primitives to be renderedin the same pass are organized into lists in a diagonal fashion at 805.A shader is run on the subblocks with the same number “X” at 806. Thesubblocks are processed on the hardware in parallel using the sameshader, and the only limitation on the amount of data processed at onetime is the amount of available hardware.

At 808, it is determined whether number “X” is the last number among thenumbers designating subblocks yet to be processed. If “X” is not thelast number, the process returns to 806 to run the shader on subblockswith a new number “X”. If “X” is the last number, then the frame isready for the deblocking operation.

Deblocking Filtering

After inter-prediction and intra-prediction are completed for the entireframe, the frame is an image without any “holes” or “garbage”. The edgesbetween and inside macro blocks are filtered with a deblocking filter toease the transition that results from decoding on a macro block level.FIG. 9 is a block diagram of a frame 902 after inter-prediction andintra-prediction have been performed. FIG. 9 illustrates the deblockinginterdependency among macro blocks. Some of the macro blocks in frame902 are shown and numbered. Each macro block depends on its neighboringleft and top macro blocks, meaning these left and top neighbors must bedeblocked first. For example, macro block 0 has no dependencies on othermacro blocks. Macro blocks 1 each depend on macro block 0, and so on.Each similarly numbered macro block has similar interdependencies.Embodiments of the invention exploit this arrangement by recognizingthat all of the similar macro blocks can be rendered in parallel. In anembodiment, each diagonal strip is rendered in a separate pass. Thedeblocking operation moves through the frame 902 to the right and downas shown by the arrows in FIG. 9.

FIGS. 10A and 10B are block diagrams of macro blocks illustratingvertical and horizontal deblocking, which are performed on each macroblock. FIG. 10A is a block diagram of a macro block 1000 that shows howvertical deblocking is arranged. Macro block 1000 is 16×16 pels, aspreviously defined. This includes 16×4 pixels as pixels are defined inan embodiment. The numbered dashed lines 0, 1, 2, and 3 designatevertical edges to be deblocked. In other embodiment there may be more orless pels per pixel, for example depending on a GPU architecture.

FIG. 10B is a block diagram of the macro block 1000 that shows howhorizontal deblocking is arranged. The numbered dashed lines 0, 1, 2,and 3 designate horizontal edges to be deblocked.

FIGS. 11A, 11B, 11C, and 11D show the pels involved in verticaldeblocking for each vertical edge in the macro block 1000. In FIG. 11A,the shaded pels, including pels from a previous (left neighboring) macroblock are used in the deblocking operation for edge 0.

In FIG. 11 b, the shaded pels on either side of edge 1 are used in avertical deblocking operation for edge 1.

In FIG. 11C, the shaded pels on either side of edge 2 are used in avertical deblocking operation for edge 2.

In FIG. 11D, the shaded pels on either side of edge 3 are used in avertical deblocking operation for edge 3.

FIGS. 12A, 12B, 12C, and 12D show the pels involved in horizontaldeblocking for each horizontal edge in the macro block 1000. In FIG.12A, the shaded pels, including pels from a previous (top neighboring)macro block are used in the deblocking operation for edge 0.

In FIG. 12 b, the shaded pels on either side of edge 1 are used in ahorizontal deblocking operation for edge 1.

In FIG. 12C, the shaded pels on either side of edge 2 are used in ahorizontal deblocking operation for edge 2.

In FIG. 12D, the shaded pels on either side of edge 3 are used in ahorizontal deblocking operation for edge 3.

In an embodiment, the pels to be processed in the deblocking algorithmare copied to a scratch buffer (for example, see FIG. 3) in order tooptimally arrange the pel data to be processed for a particular graphicsprocessing, or video processing architecture. A unit of data on whichthe hardware operates is referred to as a “quad”. In an embodiment, aquad is 2×2 pixels, where a pixel is meant as a “hardware pixels”. Ahardware pixel can be 2×2 of 4×4 pels, 8×8 pels, or 2×2 of ARGB pixels,or others arrangements. In an embodiment, the data to be processed inhorizontal deblocking and vertical deblocking is first remapped onto aquad structure in the scratch buffer. The deblocking processing isperformed and the result is written to the scratch buffer, then back tothe frame in the appropriate location. In the example architecture, thepels are grouped to exercise all of the available hardware. The pels tobe processed together may come from anywhere in the frame as long as themacro blocks from which they come are all of the same type. Having thesame type means having the same macro block dependencies. The use of aquad as a unit of data to be processed and the processing of four quadsat one time are just one example of an implementation. The sameprinciples applied in rearranging the pel data for processing can beapplied to any different graphics processing architecture.

In an embodiment, deblocking is performed for each macro block startingwith a vertical pass (vertical edge 0, vertical edge 1, vertical edge 2,vertical edge 3) and then a horizontal pass (horizontal edge 0,horizontal edge 1, horizontal edge 2, horizontal edge 3). Theparallelism inherent in the hardware design is exploited by processingmacro blocks that have no dependencies (also referred to as beingindependent) together. According to various embodiments, any number ofindependent macro blocks at may be processed at the same time, limitedonly by the hardware.

FIGS. 13-19 are block diagrams that illustrate mapping to the scratchbuffer according to an embodiment. These diagrams are an example ofmapping to accommodate a particular architecture and are not intended tobe limiting.

FIG. 13A is a block diagram of a macro block that shows vertical edges0-3. The shaded area represents data involved in a deblocking operationfor edges 0 and 1, including data (on the far left) from a previousmacro block. FIG. 13B is a block diagram that shows the conceptualmapping of the shaded data from FIG. 13A into the scratch buffer. In anembodiment, there are three scratch buffers that allow 16×3 pixels tofit in an area of 4×4 pixels, but other embodiments are possible withinthe scope of the claims. In an embodiment deblocking mapping allowsoptimal use of four pipelines (Pipe 0, Pipe 1, Pipe 2, and Pipe 3) inthe example architecture that has been previously described herein.However, the concepts described with reference to specific examplearchitectures are equally applicable to other architectures notspecifically described. For example, deblocking as described is alsoapplicable or adaptable to future architectures (for example, 8×8 or16×16) in which the screen tiling may not really exist.

FIG. 14A is a block diagram that shows multiple macro blocks and theiredges. Each of the macro blocks is similar to the single macro blockshown in FIG. 13A. FIG. 14A shows the data involved in a single verticaldeblocking pass according to an embodiment. FIG. 14B is a block diagramthat shows the mapping of the shaded data from FIG. 14A into the scratchbuffer in an arrangement that optimally uses the available hardware.

FIG. 15A is a block diagram of a macro block that shows horizontal edges0-3. The shaded area represents data involved in a deblocking operationfor edge 0, including data (at the top) from a previous macro block.FIG. 15B is a block diagram that shows the conceptual mapping of theshaded data from FIG. 15A into the scratch buffer in an arrangement thatoptimally uses available pipelines in the example architecture that hasbeen previously described herein.

FIG. 16A is a bock diagram that shows multiple macro blocks and theiredges. Each macro block is similar to the single macro block shown inFIG. 15A. The shaded data is the data involved in deblocking for edges0. FIG. 16B is a block diagram that shows the mapping of the shaded datafrom FIG. 16A into the scratch buffer in an arrangement that optimallyuses the available hardware for performing deblocking on edges 0.

FIG. 17A is a bock diagram that shows multiple macro blocks and theiredges. The shaded data is the data involved in deblocking for edges 1.FIG. 17B is a block diagram that shows the mapping of the shaded datafrom FIG. 17A into the scratch buffer in an arrangement that optimallyuses the available hardware for performing deblocking on edges 1.

FIG. 18A is a bock diagram that shows multiple macro blocks and theiredges. The shaded data is the data involved in deblocking for edges 2.FIG. 18B is a block diagram that shows the mapping of the shaded datafrom FIG. 18A into the scratch buffer in an arrangement that optimallyuses the available hardware for performing deblocking on edges 2.

FIG. 19A is a bock diagram that shows multiple macro blocks and theiredges. The shaded data is the data involved in deblocking for edges 3.FIG. 19B is a block diagram that shows the mapping of the shaded datafrom FIG. 19A into the scratch buffer in an arrangement that optimallyuses the available hardware for performing deblocking on edges 3.

The mapping shown in FIGS. 13-19 is just one example of a mapping schemefor rearranging the pel data to be processed in a manner that optimizesthe use of the available hardware.

Other variations on the methods and apparatus as described are alsowithin the scope of the invention as claimed. For example, a scratchbuffer could also be used in the inter-prediction and/orintra-prediction operations. Depending upon various factors, includingthe architecture of the graphics processing unit, using a scratch buffermay or may not be more efficient than processing “in place”. In theembodiments described, which refer a particular architecture for thepurpose of providing a coherent explanation, the deblocking operationbenefits from using the scratch buffer. One reason is that the size andconfiguration of the pel data to be processed and the number ofprocessing passes required do not vary. In addition, the order of thecopies can vary. For example, copying can be done after every diagonalor after all of the diagonals. Therefore, the rearrangement for aparticular architecture does not vary, and any performance penaltiesrelated to copying to the scratch buffer and copying back to the framecan be calculated. These performance penalties can be compared to theperformance penalties associated with processing the pel data in place,but in configurations that are not optimized for the hardware. Aninformed choice can then be made regarding whether to use the scratchbuffer or not. On the other hand, for intra-prediction for example, theunits of data to be processed are randomized by the encoding process, soit is not possible to accurately predict gains or losses associated withusing the scratch buffer, and the overall performance over time may beabout the same as for processing in place.

In another embodiment, the deblocking filtering is performed by a vertexshader for an entire macro block. In this regard the vertex shader worksas a dedicated hardware pipeline. In various embodiments with differentnumbers of available pipelines, there may be four, eight or moreavailable pipelines. In an embodiment, the deblocking algorithm involvestwo passes. The first pass is a vertical pass for all macro blocks alongthe diagonal being filtered (or deblocked). The second pass is ahorizontal pass along the same diagonal.

The vertex shader process 256 pels of the luma macro block and 64 pelsof each chroma macro block. In an embodiment, the vertex shader passesresulting filtered pels to pixel shaders through 16 parameter registers.Each register (128 bits) keeps one 4×4 filtered block of data. The“virtual pixel”, or the pixel visible to the scan converter is an 8×8block of pels for most of the passes. In an embodiment, eight rendertargets are defined. Each render target has a pixel format with twochannels, and 32 bits per channel.

The pixel shader is invoked per 8×8 block. The pixel shader selects fourproper registers from the 16 provided, rearranges them into eight2×32-bit output color registers, and sends the data to the color buffer.In an embodiment, two buffers are used, a source buffer, and a targetbuffer. For this discussion, the target buffer is the scratch buffer.The source buffer is used as texture and the target is comprised ofeither four or eight render targets. The following tables illustratebuffer states during deblocking.

FIGS. 20 and 21 show the state of the source buffer (FIG. 20) and thetarget buffer (FIG. 21) at the beginning of an algorithm iterationdesignated by the letter C. “C” marks the diagonal of the macro blocksto be filtered at the iteration C. “P” marks the previous diagonal. Bothsource buffer and target buffer keep the same data. Darkly shaded cellsindicate already filtered macro blocks, white cells indicatenot-yet-filtered macro blocks. Lightly shaded cells are partiallyfiltered in the previous iteration. The iteration C consists of severalpasses.

Pass1: Filtering the Left Side of the 0^(th) Vertical Edge of Each CMacro Block.

This pass is running along the P diagonal. Since the cell with an “X” inFIG. 21 has no right neighbor, it is not a left neighbor itself and thusit is not taking part in this pass. A peculiarity of this pass is thatthe pixel shader is invoked per 4×4 block and not per 8×8 block as in“standard” mode. 16 parameter registers are still sent to the pixelshader, but they are unpacked 32 bit float values. The target in thiscase has an ARGB type pixel format. There are 4 render targets. FIG. 22shows the state of the target buffer after the left side filtering.

Pass2: Filtering Vertical Edges of Each C Macro Block.

This pass is running along the C diagonal. During this pass thevertex/pixel shader pair is in a standard mode of operation. That is,the vertex shader sends 16 registers keeping a packed block of 4×4 pelseach, and the pixel shader is invoked per 8×8 block, target pixel format(2 channel, 32 bit per channel). There are 8 render targets. FIG. 23shows the state of the target after the vertical filtering. After pass2the source and target are switched.

Pass3: Copying the State of the P Diagonal Only from the New Source (OldTarget) to the New Target (Old Source).

FIG. 23 is a new source now. FIG. 24 presents the state of the newtarget after the copy. In this pass the vertex shader does nothing. Thepixel shader copies texture pixels in standard mode (format: 2 channels,32 per channel, virtual pixel is 8×8) directly into the frame buffer. 8render targets are involved.

Pass4: Filtering the Up Side of the 0^(th) Horizontal Edge of Each CMacro Block.

This pass is running along the P diagonal. Since the cell with an “X” inFIG. 24 has no down neighbor it is not an up neighbor itself and thus itis not taking part in the pass. FIG. 25 represents the target stateafter the pass. It shows that the P diagonal is fully filtered insidethe target frame buffer. The vertex/pixel shader pair works in the samemode as in pass 1.

Pass5: Filtering Horizontal Edges of Each C Macro Block.

This pass is running along the C diagonal. The resulting target is shownin FIG. 26. Notice that, since the horizontal filter has been applied tothe vertically filtered pels from the source (FIG. 23), the target Ccells are now both vertically and horizontally filtered.

After pass2 the source and target are switched.

Pass6: Copying the State of the P and C Diagonals from the New Source(Old Target) to the New Target (Old Source).

FIG. 26 is a now source. FIG. 23 is a new target. FIG. 27 is the stateof the target after copy. The copying is done the same way as describedwith reference to Pass3.

After making P=C, and C=C+1, the algorithm is ready for the nextiteration.

Aspects of the embodiments described above may be implemented asfunctionality programmed into any of a variety of circuitry, includingbut not limited to programmable logic devices (PLDs), such as fieldprogrammable gate arrays (FPGAs), programmable array logic (PAL)devices, electrically programmable logic and memory devices, andstandard cell-based devices, as well as application specific integratedcircuits (ASICs) and fully custom integrated circuits. Some otherpossibilities for implementing aspects of the embodiments includemicrocontrollers with memory (such as electronically erasableprogrammable read only memory (EEPROM)), embedded microprocessors,firmware, software, etc. Furthermore, aspects of the embodiments may beembodied in microprocessors having software-based circuit emulation,discrete logic (sequential and combinatorial), custom devices, fuzzy(neural) logic, quantum devices, and hybrids of any of the above devicetypes. Of course the underlying device technologies may be provided in avariety of component types, e.g., metal-oxide semiconductor field-effecttransistor (MOSFET) technologies such as complementary metal-oxidesemiconductor (CMOS), bipolar technologies such as emitter-coupled logic(ECL), polymer technologies (e.g., silicon-conjugated polymer andmetal-conjugated polymer-metal structures), mixed analog and digital,etc.

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense as opposed to anexclusive or exhaustive sense; that is to say, in a sense of “including,but not limited to.” Words using the singular or plural number alsoinclude the plural or singular number, respectively. Additionally, thewords “herein,” “hereunder,” “above,” “below,” and words of similarimport, when used in this application, refer to this application as awhole and not to any particular portions of this application. When theword “or” is used in reference to a list of two or more items, that wordcovers all of the following interpretations of the word, any of theitems in the list, all of the items in the list, and any combination ofthe items in the list.

The above description of illustrated embodiments of the method andsystem is not intended to be exhaustive or to limit the invention to theprecise forms disclosed. While specific embodiments of, and examplesfor, the method and system are described herein for illustrativepurposes, various equivalent modifications are possible within the scopeof the invention, as those skilled in the relevant art will recognize.The teachings of the disclosure provided herein can be applied to othersystems, not only for systems including graphics processing or videoprocessing, as described above. The various operations described may beperformed in a very wide variety of architectures and distributeddifferently than described. In addition, though many configurations aredescribed herein, none are intended to be limiting or exclusive.

In other embodiments, some or all of the hardware and softwarecapability described herein may exist in a printer, a camera,television, a digital versatile disc (DVD) player, a handheld device, amobile telephone or some other device. The elements and acts of thevarious embodiments described above can be combined to provide furtherembodiments. These and other changes can be made to the method andsystem in light of the above detailed description.

In general, in the following claims, the terms used should not beconstrued to limit the method and system to the specific embodimentsdisclosed in the specification and the claims, but should be construedto include any processing systems and methods that operate under theclaims. Accordingly, the method and system is not limited by thedisclosure, but instead the scope of the method and system is to bedetermined entirely by the claims.

While certain aspects of the method and system are presented below incertain claim forms, the inventors contemplate the various aspects ofthe method and system in any number of claim forms. For example, whileonly one aspect of the method and system may be recited as embodied incomputer-readable medium, other aspects may likewise be embodied incomputer-readable medium. Accordingly, the inventors reserve the rightto add additional claims after filing the application to pursue suchadditional claim forms for other aspects of the method and system.

What is claimed is:
 1. A video data decoding method comprising:pre-processing control maps generated from encoded video data that wasencoded according to a pre-defined format, wherein pre-processingcomprises generating a plurality of intermediate control maps for arespective plurality of frame processing operations containing controlinformation, the control information including an indication of whichmacro blocks or portions of macro blocks of a frame may be processed inparallel in respective frame processing operations such that oneintermediate control map can control frame processing for aninter-prediction algorithm where one set of macro blocks of a frame areidentified for parallel processing and another intermediate control mapcan control frame processing for an intra-prediction algorithm where anentirely different set of macro blocks of the frame are identified forparallel processing; and wherein the pre-defined format comprises acompression scheme according to which the video data may be encodedusing one of a plurality of prediction operations for various units ofvideo data in a frame, the plurality of prediction operations comprisinginter-prediction, wherein the plurality of intermediate control maps andat least one buffer are generated by running a pre-shader on the controlmaps based on at least one predetermined value that is set to indicatewhether particular macro blocks are interprediction, intraprediction orboth interprediction and intraprediction, the at least on buffercontaining a subset of control information indicating which of the macroblocks are interprediction, intraprediction or both interprediction andintraprediction; determining from an intermediate control map indicatedunits of video data that are to be decoded using inter-prediction; andperforming inter-prediction on all of the indicated units of video datain the frame in parallel in a respective frame processing operation. 2.The method of claim 1, further comprising performing inter-prediction onall of the indicated video data in multiple interleaved frames inparallel.
 3. The method of claim 1, further comprising parallelprocessing using the intermediate control maps and a plurality ofprocessing pipelines.
 4. The method of claim 3, wherein the plurality ofprocessing pipelines comprise a plurality of graphics processing unit(GPU) pipelines.
 5. The method of claim 1, wherein the controlinformation comprises a rearrangement of the video data such that adecoding operation can be performed in parallel on multiple video datausing the plurality of GPU pipelines.
 6. The method of claim 1, whereinthe buffer is a Z-buffer.
 7. The method of claim 6, wherein determiningcomprises Z-testing to determine which of the plurality of predictionoperations to perform on a unit of video data.
 8. The method of claim 1,wherein the compression scheme comprises one of a plurality ofhigh-compression-ratio schemes, including H.264.
 9. The method of claim1, wherein the pre-defined format comprises an MPEG standard videoformat.
 10. The method of claim 1, wherein performing inter-predictionon all of the indicated units of video data comprises broadcastinginformation from the intermediate control maps to each indicated unit ofvideo data.
 11. The method of claim 10, further comprising finding areference frame using the information.
 12. The method of claim 11,further comprising finding reference pels within the reference frameusing the information.
 13. The method of claim 12, further comprisingcombining reference pel data and residual data.
 14. The method of claim13, further comprising writing a result for each indicated unit of datato a partially decoded frame.
 15. A digital image generated by themethod of claim
 1. 16. A method for decoding video data encoded using ahigh-compression-ratio codec, the method comprising: pre-processingcontrol maps that were generated during encoding of the video data; andgenerating a plurality of intermediate control maps for a respectiveplurality of frame processing operations comprising informationincluding an indication of which macro blocks or portions of macroblocks may be processed in parallel in respective frame processingoperations, the information also including information regardingperforming inter-prediction on the video data on a frame basis such thatinter-prediction is performed on an entire frame at one time in arespective frame processing operation such that one intermediate controlmap can control frame processing for an inter-prediction algorithm whereone set of macro blocks of a frame are identified for parallelprocessing and another intermediate control map can control frameprocessing for an intra-prediction algorithm where an entirely differentset of macro blocks of the frame are identified for parallel processing,wherein the plurality of intermediate control maps and at least onebuffer are generated by running a pre-shader on the control maps basedon at least one predetermined value that is set to indicate whetherparticular macro blocks are interprediction, intraprediction or bothinterprediction and intraprediction, the at least on buffer containing asubset of control information indicating which of the macro blocks areinterprediction, intraprediction or both interprediction andintraprediction.
 17. The method of claim 16, wherein the at least onebuffer is a Z-buffer and further comprising executing a plurality ofsetup passes on the control maps, comprising performing Z-testing of theZ-buffer created from the control maps, wherein at least one Z-buffertest indicates which of the units of video data to performinter-prediction on.
 18. A non-transitory computer readable mediumincluding instructions which when executed in a video processing systemcause the system to decode video data, the decoding comprising:pre-processing control maps generated from encoded video data that wasencoded according to a pre-defined format, wherein pre-processingcomprises generating a plurality of intermediate control maps for arespective plurality of frame processing operations containing controlinformation, the control information including an indication of whichmacro blocks or portions of macro blocks may be processed in parallel inrespective frame processing operations such that one intermediatecontrol map can control frame processing for an inter-predictionalgorithm where one set of macro blocks of a frame are identified forparallel processing and another intermediate control map can controlframe processing for an intra-prediction algorithm where an entirelydifferent set of macro blocks of the frame are identified for parallelprocessing, and wherein the pre-defined format comprises a compressionscheme according to which the video data may be encoded using one of aplurality of prediction operations for various units of video data in aframe, the plurality of prediction operations comprisinginter-prediction, wherein the plurality of intermediate control maps andat least one buffer are generated by running a pre-shader on the controlmaps based on at least one predetermined value that is set to indicatewhether particular macro blocks are interprediction, intraprediction orboth interprediction and intraprediction, the at least on buffercontaining a subset of control information indicating which of the macroblocks are interprediction, intraprediction or both interprediction andintraprediction; determining from an intermediate control map indicatedunits of video data are to be decoded using inter-prediction; andperforming inter-prediction on all of the indicated units of video datain the frame in parallel in a respective frame processing operation. 19.The non-transitory computer readable medium of claim 18, wherein thedecoding further comprises performing inter-prediction on all of theindicated video data in multiple interleaved frames in parallel.
 20. Thenon-transitory computer readable medium of claim 18, wherein thedecoding further comprises parallel processing using the intermediatecontrol maps and a plurality of processing pipelines.
 21. Thenon-transitory computer readable medium of claim 20, wherein theplurality of processing pipelines comprise a plurality of graphicsprocessing unit (GPU) pipelines.
 22. The non-transitory computerreadable medium of claim 18, wherein the control information comprises arearrangement of the video data such that a decoding operation can beperformed in parallel on multiple video data using the plurality of GPUpipelines.
 23. The non-transitory computer readable-medium of claim 18,wherein determining comprises Z-testing to determine which of theplurality of prediction operations to perform on a unit of video data.24. The non-transitory computer readable medium of claim 18, wherein thecompression scheme comprises one of a plurality ofhigh-compression-ratio schemes, including H.264.
 25. The non-transitorycomputer readable medium of claim 18, wherein the pre-defined formatcomprises an MPEG standard video format.
 26. The non-transitory computerreadable medium of claim 18, wherein performing inter-prediction on allof the indicated units of video data comprises broadcasting informationfrom the intermediate control maps to each indicated unit of video data.27. The non-transitory computer readable medium of claim 26, whereinperforming inter-prediction further comprises finding a reference frameusing the information.
 28. The non-transitory computer readable mediumof claim 27, wherein performing inter-prediction further comprisesfinding reference pels within the reference frame using the information.29. The non-transitory computer readable medium of claim 28, whereinperforming inter-prediction further comprises combining reference peldata and residual data.
 30. The non-transitory computer readable mediumof claim 29, wherein performing inter-prediction further compriseswriting a result for each indicated unit of data to a partially decodedframe.
 31. A non-transitory computer readable medium having instructionsstored thereon which, when processed, are adapted to create a circuitcapable of performing a video data decoding method comprising:pre-processing control maps generated from encoded video data that wasencoded according to a pre-defined format, wherein pre-processingcomprises generating a plurality of intermediate control maps for arespective plurality of frame processing operations containing controlinformation, the control information including an indication of whichmacro blocks or portions of macro blocks may be processed in parallel inrespective frame processing operations such that one intermediatecontrol map can control frame processing for an inter-predictionalgorithm where one set of macro blocks of a frame are identified forparallel processing and another intermediate control map can controlframe processing for an intra-prediction algorithm where an entirelydifferent set of macro blocks of the frame are identified for parallelprocessing, and wherein the pre-defined format comprises a compressionscheme according to which the video data may be encoded using one of aplurality of prediction operations for various units of video data in aframe, the plurality of prediction operations comprisinginter-prediction, wherein the plurality of intermediate control maps andat least one buffer are generated by running a pre-shader on the controlmaps based on at least one predetermined value that is set to indicatewhether particular macro blocks are interprediction, intraprediction orboth interprediction and intraprediction, the at least on buffercontaining a subset of control information indicating which of the macroblocks are interprediction, intraprediction or both interprediction andintraprediction; determining from an intermediate control map indicatedunits of video data that are to be decoded using inter-prediction;performing inter-prediction on all of the indicated units of video datain the frame in parallel in a respective frame processing operation. 32.A computer having instructions stored thereon which, when implemented ina video processing driver, cause the driver to perform a parallelprocessing method, the method comprising: pre-processing control mapsthat were generated from encoded video data; and generating intermediatecontrol maps for a respective plurality of frame processing operationscomprising information including an indication of which macro blocks orportions of macro blocks may be processed in parallel in respectiveframe processing operations, the information also including informationregarding decoding the video data on a frame basis such that aninter-prediction operation is performed on an entire frame at one timein a respective frame processing operation such that one intermediatecontrol map can control frame processing for an inter-predictionalgorithm where one set of macro blocks of a frame are identified forparallel processing and another intermediate control map can controlframe processing for an intra-prediction algorithm where an entirelydifferent set of macro blocks of the frame are identified for parallelprocessing, wherein the plurality of intermediate control maps and atleast one buffer are generated by running a pre-shader on the controlmaps based on at least one predetermined value that is set to indicatewhether particular macro blocks are interprediction, intraprediction orboth interprediction and intraprediction, the at least on buffercontaining a subset of control information indicating which of the macroblocks are interprediction, intraprediction or both interprediction andintraprediction.
 33. A graphics processing unit (GPU) configured toperform motion compensation, comprising: pre-processing control mapsthat were generated from encoded video data; generating intermediatecontrol maps for a respective plurality of frame processing operationsthat indicate which macro blocks or portions of macro blocks may beprocessed in parallel in respective frame processing operations andwhich units of video data in a frame are to be processed using aninter-prediction operation such that one intermediate control map cancontrol frame processing for an inter-prediction algorithm where one setof macro blocks of a frame are identified for parallel processing andanother intermediate control map can control frame processing for anintra-prediction algorithm where an entirely different set of macroblocks of the frame are identified for parallel processing, wherein theplurality of intermediate control maps and at least one buffer aregenerated by running a pre-shader on the control maps based on at leastone predetermined value that is set to indicate whether particular macroblocks are interprediction, intraprediction or both interprediction andintraprediction, the at least on buffer containing a subset of controlinformation indicating which of the macro blocks are interprediction,intraprediction or both interprediction and intraprediction; and usingan intermediate control map in a respective frame processing operationto perform inter-prediction on the video data on a frame basis such thateach inter-prediction is performed on an entire frame at one time, andto further rearrange the video data to be processed in parallel onmultiple pipelines of the GPU.
 34. A video processing apparatuscomprising: circuitry configured to pre-process control maps that weregenerated from encoded video data that was encoded according to apredefined format, and to generate intermediate control maps for arespective plurality of frame processing operations that indicate whichmacro blocks or portions of macro blocks may be processed in parallel inrespective frame processing operations and which units of video data ina frame are to be processed using an inter-prediction operation suchthat one intermediate control map can control frame processing for aninter-prediction algorithm where one set of macro blocks of a frame areidentified for parallel processing and another intermediate control mapcan control frame processing for an intra-prediction algorithm where anentirely different set of macro blocks of the frame are identified forparallel processing, wherein the plurality of intermediate control mapsand at least one buffer are generated by running a pre-shader on thecontrol maps based on at least one predetermined value that is set toindicate whether particular macro blocks are interprediction,intraprediction or both interprediction and intraprediction, the atleast on buffer containing a subset of control information indicatingwhich of the macro blocks are interprediction, intraprediction or bothinterprediction and intraprediction; and driver circuitry configured toread the intermediate control maps for controlling a video data decodingoperation, including performing the inter-prediction operation based onan intermediate control map in a respective frame processing operation;and multiple video processing pipeline circuitry configured to respondto the driver circuitry to perform decoding of the video data on a framebasis such that the inter-prediction is performed on an entire frame atone time, and to further rearrange the video data to be processed inparallel on multiple pipelines of the multiple video processing pipelinecircuitry.
 35. A method for decoding video data, comprising: a firstprocessor generating control maps from encoded video data; a secondprocessor, receiving the control maps; generating intermediate controlmaps for a respective plurality of frame processing operations from thecontrol maps such that one intermediate control map can control frameprocessing for an inter-prediction algorithm where one set of macroblocks of a frame are identified for parallel processing and anotherintermediate control map can control frame processing for anintra-prediction algorithm where an entirely different set of macroblocks of the frame are identified for parallel processing, wherein theplurality of intermediate control maps and at least one buffer aregenerated by running a pre-shader on the control maps based on at leastone predetermined value that is set to indicate whether particular macroblocks are interprediction, intraprediction or both interprediction andintraprediction, the at least on buffer containing a subset of controlinformation indicating which of the macro blocks are interprediction,intraprediction or both interprediction and intraprediction, and whereinone of the intermediate control maps indicates: which units of videodata in a frame are to be processed using an inter-prediction operation,and; which macro blocks or portions of macro blocks may be processed inparallel; and using the one intermediate control map to decode theencoded video data in a respective frame processing operation,comprising performing inter-prediction on all of the indicated units inthe frame in parallel.
 36. The method of claim 35, wherein theintermediate control maps further comprise information specific to anarchitecture of the second processor.
 37. The method of claim 36,further comprising the second processor using the intermediate controlmaps to perform parallel processing on the video data to generatedisplay data.
 38. The method of claim 37, wherein parallel processingcomprises performing set up passes.
 39. The method of claim 38, whereinperforming setup passes comprises at least one of: sorting passes tosort surfaces; inter-prediction passes; and intra-prediction passes. 40.The method of claim 36, wherein control maps are generated on a perframe basis.
 41. The method of claim 36, wherein the architecture of thesecond processor comprises a type of architecture selected from a groupcomprising: a single instruction multiple data (SIMD) architecture; amulti-core architecture; and a multi-pipeline architecture.
 42. Themethod of claim 35, wherein the control maps comprise data and controlinformation according to a specified video encoding format.